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Abstract 

Increasing  digitization  of  the  modern  battlefield  brings  with  it  new  demands  on  military 
command  staffs  for  rapid,  flexible  decision  making,  execution  of  complex  digital  skills  such  as 
visualization  and  information  management,  and  coordinated,  communication-intensive 
teamwork.  The  Army's  Battle-Command  Reengineering  III  exercise  (BCR  III)  simulated  the 
demands  of  the  future  digital  command  environment  by  placing  a  command  staff  in  the  context 
of  a  soldier-in-the-loop  simulation,  with  a  suite  of  digital  tools  including  email,  shared 
whiteboard,  and  SA-enhancing  common-operating  views,  to  test  concepts  in  future  battle 
command.  In  this  context,  we  developed  an  innovative  approach  to  decision-  skills  training  using 
short,  focused  vignettes  that  provided  staff  members  with  opportunities  to  utilize  the  same  digital 
equipment  to  accomplish  goals  in  each  of  several  stages  of  the  natural  battle-preparation  and 
execution  cycle.  To  assess  the  degree  to  which  the  training  approach  was  successful  and  to 
develop  a  picture  of  individual  and  team  outcome/process,  we  developed  a  set  of  targeted 
performance  measures.  These  measures  fell  into  several  categories  ranging  from  measures  of 
individual  performance  to  measures  of  team- level  perceptions  of  teammates’  roles  and  workload. 
We  employed  a  mix  of  observation,  participant  survey  and  objective  measurement  to  provide 
measures  of  both  team  outcome  and  team  processes. 
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1  Introduction  and  Significance  to  Command  and  Control 


Increasing  digitization  of  the  modern  battlefield  brings  with  it  new  demands  on  military 
command  staffs  for  rapid,  flexible  decision  making,  execution  of  complex  digital  skills  such  as 
visualization  and  information  management,  and  coordinated,  communication-intensive 
teamwork.  The  same  technologies  that  are  driving  battlefield  digitization  are  also  driving  a 
revolution  in  command  training  and  performance  assessment:  It  is  now  possible,  using  high- 
fidelity  distributed  simulations,  to  expose  command  teams  to  realistic,  rigorous  simulated 
command  exercises  that  provide  practice  on  a  range  of  cognitive  skills  and  demand  considerable 
coordination  of  action.  Indeed  -and  consistent  with  John  Shalikashvili's  Joint  Vision  2010  mew 
high-fidelity  simulation  technologies  are  making  it  increasingly  possible  for  today's  command 
staffs  to  "train  the  way  [they]  fight."  These  new  capabilities  have  brought  with  them  a  number  of 
new  challenges  in  training  and  performance  assessment,  however,  and  innovations  in  training 
and  assessment  must  now  catch  up  with  the  pace  of  technological  innovation  to  insure  the 
benefits  of  simulation-based  training. 

The  introduction  of  digital  technologies  and  the  increasing  volume  of  information  these 
technologies  have  made  possible  have  led  to  a  shifting  emphasis  in  the  training  needed  by  today's 
commanders.  Increasingly,  digital  skills  such  as  information  management,  rapid  visualization, 
and  information  fusion  are  becoming  required  on  the  battlefield.  Teamwork  skills  have  also 
become  more  important  in  the  era  of  network-centric  warfare  (NCW)  where  teams  of  specialists 
are  tethered  together  by  satellites,  and  coordination  is  accomplished  through  virtual  collaboration 
and  email.  In  addition  to  the  need  to  modernize  our  approaches  to  training,  traditional  measures 
of  performance  and  effectiveness  (MOPs  and  MOEs)  must  also  be  adapted  to  meet  the 
challenges  of  measuring  performance  in  these  new  skill  domains  and  assessing  digital-skill 
learning  and  retention.  Measures  must  capture  both  outcome  (were  the  right  decision  made?)  and 
process  (were  the  decisions  made  right?)  to  effectively  assess  command  performance  in  digital 
environments. 

The  Army's  Battle  Command  Reengineering  III  exercise  (BCR  III),  took  place  at  the  Mounted 
Maneuver  Battlespace  Lab's  (MMBL's)  Mounted  Warfare  Testbed  (MWTB)  in  Fort  Knox,  KY, 
in  April  '99.  This  exercise  offered  opportunities  to  examine  new  approaches  to  training  and 
assessment  in  the  context  of  a  large-scale  digital  exercise.  The  exercise  applied  soldier-in-the- 
loop  simulation  and  semi-automated  force  technologies  to  test  concepts  in  future  battle 
command.  It  placed  a  new  command-team  architecture  in  a  digital  command  environment, 
providing  staff  control  over  digital  resources  to  support  planning,  battle  rehearsal,  coordination, 
communication,  and  situation  awareness.  Command  staff  participated  in  nine  days  of  exercises 
designed  to  assess  the  impacts  of  digital  technologies  on  command  efficiency  and  the  command 
decision-making  process.  To  prepare  participating  command  staff  for  this  rigorous  exercise,  we 
developed  decision-centered  training  based  on  focused  training  vignettes  that  emphasized  three 
key  skill  areas  -digital  decision  skills,  information  management,  and  team  performances— n  d  a 
range  of  measures  designed  to  assess  individual  and  team  performance,  process  and  outcome  in 
each  of  these  areas.  The  paragraphs  below  describe  the  approach  used  and  some  outcomes  of  the 
exercise. 


2  Training  Key  Command-Team  Skills 


The  competing  goals  of  training  individuals  to  proficiently  utilize  digital  equipment!— o  use  the 
computers,  file  systems,  email,  chat  and  white  board  capabilities  -and  training  them  to 
efficiently  command  an  ongoing  battle  using  this  equipment  -engage  in  a  "digital"  decision 
making  process  -make  the  job  of  training  digital  staff  complex.  Knowing  how  to  use  digital 
resources  in  isolation  (push  the  right  buttons)  is  not  enough  to  insure  command  efficiency  and  is 
likely  to  lead  commanders  to  fall  back  on  what  they  know  rather  than  fully  exploit  the  strengths 
of  new  technologies.  To  strike  a  balance  between  the  "push-button"  training  needed  to  achieve 
proficiency  on  the  equipment  and  the  more  decision-oriented  command-level  training  needed  to 
mesh  digital  skills  into  the  command  decision-making  process,  we  separated  these  aspects  in  our 
training. 

The  BCR  III  training  program  utilized  a  decision-centered  approach,  using  short,  focused 
vignettes  that  placed  the  command  staff  in  isolated  stages  of  the  natural  battle-preparation  and 
execution  cycle.  By  isolating  battle  planning,  course  of  action  (COA)  development,  rehearsal, 
and  execution ,  we  trained  the  staff  how  to  utilize  the  same  digital  equipment  to  accomplish  goals 
of  each  stage.  This  is  important  because  the  same  tools  -email,  whiteboard,  plan  view  display 
(PVD)  overlays,  video  teleconference,  and  stealth  display  -were  utilized  in  different  ways  in 
each  stage. 

In  addition  to  isolating  natural  stages  in  the  battle  preparation  and  execution  cycle,  we  also  built 
"themes"  into  our  vignettes  to  insure  stress  on  key  digital-command  competencies.  We  focused 
on  three  main  skill  areas:  decision  making  skills,  information  management  strategies,  and  team 
performance  strategies.  While  there  were  demands  on  each  of  these  skills  sets  in  our  vignettes, 
we  intentionally  stressed  one  over  the  others  in  individual  vignettes  in  order  to  focus  staff 
attention  on  how  the  digital  equipment  could  be  used  to  improve  each. 

Decision-centered  training  -A  igncttcs  were  intentionally  designed  to  offer  challenging 
deliberate  practice  opportunities  and  ample  feedback  and  reflection  opportunities  to  insure  that 
learning  would  take  place.  Before  each  vignette,  we  conducted  group  discussions  of  expectations 
and  roles  and  functions  designed  to  help  the  team  develop  common  mental  models  and 
strategies.  During  the  vignettes,  action  was  stopped  to  "calibrate"  situation  awareness  -get 
everyone  on  the  same  page  -and  reflect  on  performance  up  to  that  point,  correcting  anything 
that  was  not  working.  At  the  end  of  the  exercise,  the  team  critiqued  their  own  performance  and 
repaired  processes  for  the  next  vignette  (methods  based  on  Klinger  &  Klein,  1999) 

Training  teamwork  and  coordination  skills  -Research  has  show  that  effective  teams  are  able  to 
flexibly  adapt  their  decision  making  and  coordination  strategies,  and  their  overall  organizational 
structure  to  the  demands  of  performance.  Our  own  work  in  the  Tactical  Decision  Making  Under 
Stress  (TADMUS)  program  has  suggested  that  the  foundation  of  skills  required  to  adapt  in  these 
ways  can  be  trained  (Serfaty,  Entin,  &  Johnston,  1998).  In  the  current  context,  we  applied 
insights  from  Team  Adaptation  and  Coordination  Training  (TACT)  (Entin,  Serfaty,  and  Deckert, 
1994)  by  using  training  vignettes  as  opportunities  for  explicit  practice  of  proven  performance¬ 
enhancing  teamwork  strategies.  These  strategies  included:  periodic  intent  updating,  explicit 
reflection  on  team  member  roles  and  functions,  monitoring  the  workloads  of  others,  and 
anticipating  the  information  loads  and  needs  of  others. 


Training  for  digital  information  management  -Of  the  three  areas  covered,  this  one  is  perhaps  the 
most  important  to  the  digital  staff.  The  BCR  III  exercise  placed  the  command  staff  in  a  unique 
four  node  command  architecture  in  which  the  commander  was  supported  by  a  mirror-image 
deputy  command  staff  and  two  additional  nodes  concurrently  supporting  current  and  planning 
future  operations.  These  nodes  were  located  in  separate  command  vehicles  and  supported  the 
commander  by  communicating  and  sending  information  via  email,  radio,  teleconference, 
whiteboard,  and  map  overlays.  The  sheer  volume  of  data  and  number  of  data  sources  required 
that  the  staff  be  trained  to  efficiently  share  and  manage  information  to  avoid  information 
overload.  Our  training  method,  based  on  experimental  findings  and  a  related  model  of 
information  management  (Entin,  Serfaty,  Klein  &  Wolf,  1998),  trained  the  staff  to  pre-define 
information  requirements,  improve  their  level  of  organizational  awareness  (to  better  predict  the 
information  needs  of  others)  and  make  better  decision  about  when  and  how  to  pull  ( request )  and 
push  (disseminate)  information. 


3  Measuring  Team  Outcomes,  Process  and  Performance 

To  assess  team  performance,  and  provide  some  data  on  the  degree  to  which  our  training 
approach  was  successful,  we  developed  a  set  of  targeted  individual  and  team  outcome  and 
process  measures.  These  measures  fell  into  several  broad  categories  that  ranged  from  measures 
of  individual  performance  to  measures  of  team-level  perceptions  of  one  another's  roles  and 
workload.  We  employed  a  mix  of  observation,  participant  survey  and  objective  measurement. 
Importantly,  this  package  of  measures  went  beyond  typical  outcome  measures  by  looking 
directly  at  the  processes  and  performance  of  the  team.  Categories  of  measures  included: 

Outcome  performance  measures  4©  assess  dimensions  of  team  performance.  Several  of  our 
measures  were  modeled  after  event-based  performance-assessment  methods  such  as  the 
TARGETS  approach  (Fowlkes,  Lane,  Salas,  Franz,  and  Oser  ,  1994)  and  the  Advanced  Team 
Performance  Index  (ATPI)  developed  by  the  Navy  and  used  by  Serfaty,  Entin,  and  Johnston 
(1998).  These  methods  were  based  on  direct  observation  by  specialized  observers/trainers  of  pre¬ 
determined  critical  team  events  or  groups  of  events. 

Individual-level  Process  measures  -to  capture  the  mechanisms  or  processes  by  which  the  team 
achieves  performance.  These  process  measures  included  levels  of  information  seeking,  resource 
utilization,  failure  to  perform  tasks,  and  individual  decision-making  or  problem-solving 
strategies  in  the  team  context  (Serfaty  &  Kleinman,  1989). 

Team-level  process  measures  -describe  how  team  strategies  were  accomplished.  These 
observer-based  teamwork  measures  included  behaviorally-anchored  items  designed  to  assess  key 
dimensions  of  teamwork  (Serfaty,  Entin,  &  Johnston,  1998). 

Patterns  of  team  communication  -to  strike  a  balance  between  analyzing  raw  communication 
data  (e.g.,  overall  number  of  messages/unit  time)  and  an  exhaustive  semantic  analysis  of  the 
team  utterances,  we  used  a  focused  communication  capture  form.  From  the  data  collected  using 
this  observation-based  form,  more  elaborate  measures  of  team  coordination  (e.g.,  anticipation 
ratios,  information  compression  ratios,  push-pull  ratios)  can  be  computed  (Serfaty,  Entin  and 
Deckert,  1993). 

Workload  and  Cognitive  measures  -indicate  the  demands  along  various  dimensions  including 
time,  mental  effort,  and  psychological  stress.  We  utilized  variants  of  the  NASA  Task  Load  Index 


(TLX)  (Hart  and  Staveland,  1988)  to  assess  individual  and  team  workload  and  the  dynamic 
redistribution  of  workload  in  teams.  Research  has  shown  that  the  ability  to  shift  workload  among 
team  members  in  response  to  changing  task  demands  can  lead  to  more  effective  team 
performance  (e.g.,  Serfaty,  Entin,  &  Johnston,  1998).  To  assess  the  teams  abilities  to  adapt,  and 
their  perceptions  of  load  distribution,  we  utilized  several  derived  measures  of  team  workload, 
based  on  individual  team  members'  ratings  of  their  own  workload  and  the  workload  of  the  others. 

The  table  below  summarizes  the  measures  we  utilized  during  the  BCR  III  exercise. 


Performance  Measures 

Process  (How)  Outcome  (How  much) 


Individual 

♦  Individual  Workload  Index 

♦  Info-Management  Questionnaire 

♦  Organizational  Awareness  Form 

♦  Embedded  (simulation 

generated)  outcome  measures 

Team 

(Intra- 

Node)  and 

Team-of- 

T earns 

(Inter- 

Node) 

♦  Mutual  Workload  Rating 

♦  Communication  Load 
Questionnaire 

♦  Teamwork  Skills  Rating 
(process  assessment) 

♦  Operations  Planning 

Questionnaire 

♦  Embedded  (simulation 
generated)  outcome  measures 

♦  Teamwork  Skills  Rating 
(outcome) 

4  Outcomes  and  value 

The  BCR  exercise  provided  nine  full  days  of  data  collection  during  which  the  command  staff 
conducted  several,  multi-day  missions.  Observers  were  placed  at  each  of  the  four  nodes,  and 
were  able  to  listen  to  the  radio  traffic  generated  within  and  between  nodes,  as  well  as  look  over 
the  shoulders  of  staff  members  in  their  assigned  nodes  to  see  plan-view  contents  and  monitor 
white  board  sessions.  These  observers  collected  communication  data  consisting  of  counts  of 
communication  events  in  a  limited  set  of  theoretically-relevant  categories  (requests  and  transfers 
of  information,  requests  and  commitments  to  perform  actions  or  use  key  resources).  Observers 
also  provided  overall  ratings  of  team  performance  on  several  dimensions  including  back-up 
behavior,  communication,  and  coordination.  At  the  end  of  each  mission  day,  the  full  command 
staff  (14  members)  completed  a  set  of  surveys  assessing  workload,  information  management, 
team  process,  and  mission  outcome. 

Of  particular  interest  to  the  assessment  team  and  the  Battle  Lab  were  the  effects  of  information 
technologies  on  C2  performance  and  success.  As  can  be  seen  in  Figure  1,  information 
technologies  such  as  email,  video  and  whiteboard  surpassed  more  traditional  communication 
tools  such  as  radio.  This  may  have  been  due  to  the  novelty  of  the  technologies  in  this  setting, 
but,  when  workload  was  highest  and  the  team  was  attending  to  operational  goals,  these 
technologies  were  still  used,  suggesting  that  they  were  successful  communication  tools.  An 
interesting  and  related  finding,  shown  in  Figure  2,  which  is  consistent  with  what  one  would 
expect  in  a  network-centric  C2  team,  is  that  the  control  node  of  the  organization  felt  that  they 
were  spending  a  significant  amount  of  time  managing  information.  Patterns  of  information  flow 
suggest  that  the  command  node  was,  in  fact,  acting  as  a  hub  of  information  management  in  the 
organization  -and  interesting  role  that  may  suggest  an  emerging  requirement  for  future  C2 
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organizations.  The  two  bars  of  this  image  represent  self  assessment  and  the  assessment  of  the 
rest  of  the  team  as  a  check..  Same-height  bars  in  this  case  indicate  the  team  agreed  with  the 
command  node’s  assessment  of  their  own  load  and  recognized  their  information-management 
role  in  the  team.  This  is  an  important  finding,  because  evidence  suggests  that  the  ability  of 
individuals  to  appreciate  the  demands  placed  on  their  teammates  correlates  well  with  overall 
team  performance  (Serfaty,  Entin,  1995;  Serfaty,  Entin,  &  Volpe,  1993;  Serfaty,  &  Kleinman, 
1985).  These  findings  underscore  the  growing  importance  of  digital  skills  training  for  the 
network-centric  future. 

The  introduction  of  digital  technologies  and  emerging  importance  of  digital  skills,  collaborative 
work,  and  network- supported  team  coordination  are  providing  a  new  set  of  challenges  for 
training  and  performance  measurement.  Our  experience  with  the  BCR  III  has  provided  a  number 
of  lessons  learned  that  will  be  invaluable  for  future  digital  skills  training  development.  Further, 
our  experience  in  the  application  of  a  range  of  field  methods  for  capturing  and  assessing  real¬ 
time  performance  in  the  context  of  complex  mission  simulation  provides  a  solid  foundation  for 
future  performance  assessment  efforts.  Our  data  will  be  used  to  assess  the  degree  to  which  the 
staff  utilizes  digital  technologies  to  support  improved  individual  and  team  performance,  the  level 
of  awareness  individuals  maintain  of  the  workload  and  information  requirements  of  others,  and 
the  degree  to  which  our  initial  training  program  provides  an  adequate  foundation  to  orient  non¬ 
digital  staffs  to  the  possibilities  of  the  digital,  network-centric  command  environment  of  the 
future. 
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